Skip to content

WIP: Isolate ARMBUILD-Ubuntu-24.04-arm CI for failure diagnosis#5948

Open
hjmjohnson wants to merge 4 commits intoInsightSoftwareConsortium:mainfrom
hjmjohnson:isolate-arm-ci-failures
Open

WIP: Isolate ARMBUILD-Ubuntu-24.04-arm CI for failure diagnosis#5948
hjmjohnson wants to merge 4 commits intoInsightSoftwareConsortium:mainfrom
hjmjohnson:isolate-arm-ci-failures

Conversation

@hjmjohnson
Copy link
Member

@hjmjohnson hjmjohnson commented Mar 14, 2026

Summary

  • Diagnose and fix ARM CI dashboard failures caused by false-positive warnings and errors
  • Suppress GCC -Wmaybe-uninitialized warnings from ThirdParty Eigen3 via CTEST_CUSTOM_WARNING_EXCEPTION
  • Add false-positive error exceptions to CTEST_CUSTOM_ERROR_EXCEPTION for expected test output (e.g., "--- ERROR ---" banners from caught exceptions, itk::ExceptionObject traces)
  • Temporarily disable non-ARM CI pipelines to isolate the issue (WIP commit, will be reverted before merge)

Root Cause

  1. Warnings: GCC on ARM produces -Wmaybe-uninitialized warnings from Eigen3's SelfadjointMatrixVector.h. The itk_common.cmake dashboard script treats build_warnings > 0 as CI failure. These warnings are fixed upstream (Eigen commit 662d5c21f).
  2. Errors: CDash flags test output containing error-like strings (e.g., "--- ERROR ---" from expected exception handling) as build errors despite tests passing.

Commits

  • WIP: Disable all CI builds except ARMBUILD-Ubuntu-24.04-arm — CI isolation (to be reverted)
  • COMP: suppress some third party warnings on cdash — Add ThirdParty warning exceptions
  • COMP: Suppress warnings from ThirdParty Eigen module — Add Eigen-specific warning suppression
  • COMP: Add false-positive error exceptions for expected ITK test output — Suppress false-positive CDash errors

Test plan

  • Only ARMBUILD-Ubuntu-24.04-arm CI runs on this PR
  • Verify dashboard no longer reports false warnings/errors
  • Revert WIP CI isolation commit before merge

🤖 Generated with Claude Code

Temporarily disable all CI pipelines except the ARM Ubuntu build
to isolate dashboard failures for debugging.

- arm.yml: Remove x86_64-rosetta and Python matrix entries
- pixi.yml: Comment out PR trigger
- Azure pipelines: Set pr: none for Linux, LinuxPython, MacOS,
  MacOSPython, Windows, WindowsPython

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions bot added type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct labels Mar 14, 2026
@hjmjohnson
Copy link
Member Author

@blowekamp

FYI: The following build warnings only show up under the linux arm builds:

└─[0] rg "warning: " 1_ARMBUILD-Ubuntu-24.04-arm.txt|fgrep -v libpng                             (isolate-arm-ci-failures|…1⚑9)
2026-03-13T23:54:34.6885757Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-13T23:54:34.6934207Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-13T23:54:34.6951147Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:06:13.5784560Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:06:56.2395410Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:08:56.5683796Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:24:19.7212980Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:24:19.7420150Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]

and there were some changes made to EIgen yesterday.

@hjmjohnson
Copy link
Member Author

@blowekamp

A little AI evaluation of the differences for the passing and failing logs is narrowing down the issue with what look to be very plausible:

⏺ The ThirdParty patterns on lines 80-92 use a format like ".*[/\\][Mm]odules[/\\][Tt]hird[Pp]arty[/\\][Specific]..." but there's no Eigen3 entry. Lines 127 and 130 add GoogleTest and DoubleConversion entries. But the Eigen3 warnings have this path format:

../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/...

Let me check if the existing ThirdParty warning exceptions match this. The patterns on lines 80-92 are for specific subdirectories (DICOMParser, Expat, JPEG, etc.) — there's no catch-all for all of ThirdParty.

The warning is: Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]

This only appears on GCC (Ubuntu ARM) and not on Clang (macOS Rosetta), which is typical for -Wmaybe-uninitialized — it's a GCC-only warning that Clang doesn't produce.

Root cause: The ARMBUILD-Ubuntu-24.04-arm CI fails because GCC on ARM produces 8 -Wmaybe-uninitialized warnings from Eigen3's SelfadjointMatrixVector.h, which are counted by ctest_build(NUMBER_WARNINGS build_warnings). The itk_common.cmake script (line 495-498) treats any
build_warnings > 0 as a CI failure. These Eigen3 warnings are not suppressed by CTEST_CUSTOM_WARNING_EXCEPTION because there's no pattern matching Eigen3 in the exception list — unlike other ThirdParty subdirectories (JPEG, TIFF, VNL, etc.) that each have their own exception
entry.

The Rosetta build passes because Clang doesn't emit -Wmaybe-uninitialized.

@hjmjohnson
Copy link
Member Author

@blowekamp FYI:

⏺ Based on the analysis, the GCC -Wmaybe-uninitialized warnings from Eigen3's SelfadjointMatrixVector.h have been fixed upstream through a substantial rewrite of the algorithm. The warning-triggering code no longer exists in the upstream version.

@github-actions github-actions bot added the area:ThirdParty Issues affecting the ThirdParty module label Mar 14, 2026
@blowekamp
Copy link
Member

@seanm ref#5876

There are more warnings suppression here for the third party than I realized:
https://github.com/InsightSoftwareConsortium/ITK/blob/main/CMake/CTestCustom.cmake.in#L80

Adding Eigen3 here seem reasonable if that is the convention. I am not sure how the changed yesterday caused this warning to be emitted, maybe it was already being suppressed somehow.

@seanm
Copy link
Contributor

seanm commented Mar 14, 2026

how about this: #5949

@hjmjohnson hjmjohnson force-pushed the isolate-arm-ci-failures branch from e2dd0ca to 131f614 Compare March 14, 2026 23:27
@hjmjohnson hjmjohnson force-pushed the isolate-arm-ci-failures branch from 131f614 to 06fa5c6 Compare March 14, 2026 23:28
@github-actions github-actions bot removed the area:ThirdParty Issues affecting the ThirdParty module label Mar 14, 2026
Add patterns to CTEST_CUSTOM_ERROR_EXCEPTION for test output that
CDash incorrectly flags as errors. These include "--- ERROR ---"
banners from expected exception handling, itk::ExceptionObject traces,
and known expected-exception descriptions that appear in passing tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants